We introduce deep scale-spaces, a generalization of convolutional neural networks, exploiting the scale symmetry structure of conventional image recognition tasks. Put plainly, the class of an image is invariant to the scale at which it is viewed. We construct scale equivariant cross-correlations based on a principled extension of convolutions, grounded in the theory of scale-spaces and semigroups. As a very basic operation, these cross-correlations can be used in almost any modern deep learning architecture in a plug-and-play manner. We demonstrate our networks on the Patch Camelyon and Cityscapes datasets, to prove their utility and perform introspective studies to further understand their properties.

Your proposed modifications sound good. Quality of results: I mostly found the theoretical framework and the subject of scale-equivariance to be interesting, therefore I did not require the paper to achieve state-of-the-art results. The FPN paper is particularly relevant because I believe that they apply the same output function to all levels of the representation, encouraging some degree of scale-equivairance. My overall rating of the paper remains unchanged. Other comments... Related work: Another interesting multi-scale (but not scale-equivariant) paper is "convolutional neural fabrics".

This paper has resulted in a detailed discussion between the reviewers. The authors use the same techniques that have recently proved to be very popular for building neural networks that are equivariant to translations, rotations etc. to formulate a general theory of scale equivariant neural networks. I believe that the mathematical derivations are sound and that this is an interesting direction to pursue. However, at the end of the day the proposed algorithm is simple and similar to other "multi-scale" neural nets that have recently appeared in the literature. This makes one feel that the mathematical derivations are a little too meticulous, and instead the authors should have focused on conveying the intuition and providing more extensive and more convincing experimental results. The lack of the latter is particularly concerning given that comparable results published in the Vision literature are much stronger (by the authors admission they did not have the time/resources to perform similar experiments).

We introduce deep scale-spaces, a generalization of convolutional neural networks, exploiting the scale symmetry structure of conventional image recognition tasks. Put plainly, the class of an image is invariant to the scale at which it is viewed. We construct scale equivariant cross-correlations based on a principled extension of convolutions, grounded in the theory of scale-spaces and semigroups. As a very basic operation, these cross-correlations can be used in almost any modern deep learning architecture in a plug-and-play manner. We demonstrate our networks on the Patch Camelyon and Cityscapes datasets, to prove their utility and perform introspective studies to further understand their properties.

We introduce deep scale-spaces, a generalization of convolutional neural networks, exploiting the scale symmetry structure of conventional image recognition tasks. Put plainly, the class of an image is invariant to the scale at which it is viewed. We construct scale equivariant cross-correlations based on a principled extension of convolutions, grounded in the theory of scale-spaces and semigroups. As a very basic operation, these cross-correlations can be used in almost any modern deep learning architecture in a plug-and-play manner. We demonstrate our networks on the Patch Camelyon and Cityscapes datasets, to prove their utility and perform introspective studies to further understand their properties.